Image processing apparatus and method, and program to generate high-quality image using normalized image

ABSTRACT

There is provided an image processing apparatus that includes an ambient-light image obtaining section, a cumulative image generation section, and a high-quality image generation section. The ambient-light image obtaining section obtains an ambient-light image in a first time range, the ambient-light image being an image of an object captured with a predetermined exposure time. The cumulative image generation section generates a cumulative image in a second time range, the cumulative image being obtained by cumulative addition of each pixel value in a plurality of images, the plurality of images being of the object sequentially captured with the predetermined exposure time. The high-quality image generation section generates a high-quality image, the high-quality image being obtained by subtracting a pixel value in the ambient-light image from a corresponding pixel value in a normalized image, the normalized image being the cumulative image normalized based on a total sum of the exposure time.

CROSS REFERENCE TO RELATED APPLICATION(S)

This application claims the benefit of Japanese Priority PatentApplication JP 2012-246537 filed Nov. 8, 2012, the entire contents ofwhich are incorporated herein by reference.

BACKGROUND

The present technology relates to an image processing apparatus andmethod, and a program and, more specifically, to an image processingapparatus and method, and a program with which an object image iscaptured easily with less noise, balanced shading, and enhancedappearance without the use of a plurality of lighting devices.

Imaging studios with technical expertise use a multi lighting system forimaging of an object. With the multi lighting system, the object isimaged with desirable shadow created by light coming from variousdirections using a plurality of lighting devices. The use of the multilighting system enhances the appearance of the object, and leads to animage well worth viewing.

The image of an object captured using the multi lighting system looksbetter than another of the same object captured by interior lighting orusing a strobe light, for example. This is because the image is withunique shadow created by the multi lighting system, and is with thenoise reduction effect produced by the object being illuminated withsufficient level of brightness, for example.

Recently, the auction over the Internet or the like is getting verypopular. With the auction, buying and selling of goods is conductedbased on images of the goods captured by users, and the goods are notavailable for other users to check themselves. Therefore, how good theimages of the goods for sale look affects the winning price.

The demand for full-scale imaging using the multi lighting system isthus growing among the users not specifically interested in imaging.

The concern here is that imaging using the multi lighting system may notbe fully advantageous for not-experienced users because expert skill isexpected to adjust the shading balance.

In consideration thereof, a technology has been proposed to easeadjustment of the shading balance. For example, refer to Japanese PatentApplication Laid-Open No. 2007-281937 (hereinafter, referred to asPatent Document 1).

In Patent Document 1, the shading balance of an object is adjusted afterimaging thereof using a simulation image to check. That is, the objectis imaged by turning on a plurality of lighting devices one by one, andthe resulting digital images are combined by weighted addition so thatthe simulation image is obtained. This somewhat reduces the complicatedprocess of adjusting the shading balance at the time of imaging.

Also in Patent Document 1, even if the object is being illuminated bothby the ambient light and the illumination light, the problem of theresulting combined image becoming brighter than the expected image isprevented by subtraction of the ambient-light image.

Also proposed is a technology of displaying captured images at any timeduring long-exposure imaging such as bulb imaging. For example, refer toJapanese Patent Application Laid-Open No. 2005-117395 (hereinafter,referred to as Patent Document 2).

With the technology of Patent Document 2, however, overlay of aplurality of images captured by long-exposure imaging causessuperposition of noise floor. Therefore, a larger number of images perunit time during long-exposure imaging increase the noise floor in theresulting combined image.

In consideration thereof, proposed is a technology of reducing the noisefloor in the combined image by reducing the actual number of images tobe captured without reducing the display frame rate by less-frequentreading of lines. For example, refer to Japanese Patent ApplicationLaid-Open No. 2012-80457 (hereinafter, referred to as Patent Document3).

SUMMARY

The concern here is that, with the apparatus of Patent Document 1, theuser may be indeed allowed to adjust the luminance of light from thelighting devices by simulation after finishing imaging, but not allowedto adjust the position and direction of the lighting devices bysimulation, for example. This means that the previous technology hasmany restrictions on shading adjustment of the object.

Moreover, in order to use the technology of Patent Document 1, the useof a plurality of computer-controllable lighting devices is expected sothat the time and cost for the placement thereof is a burden, forexample.

In the first place, the multi lighting system itself is a large-scaleproject for amateur users with no expertise. Further, with the multilighting system, a larger number of lighting devices indeed leads to ahigher flexibility but results in more complicated shading control.

Still further, in order to reduce the effect caused by superposition ofnoise on the image such as light shot noise, the use of a lightingdevice that emits relatively strong light is expected. The use of themulti lighting system for imaging is not thus easy for the amateurusers.

Also in Patent Document 1, adequate measures are not taken against theshot noise in ambient light components. There thus is a possibility ofcausing strong noise in any portion especially dark in the image, whichis free from the ambient light components after the subtraction processperformed thereon.

This is because a pixel value a+b being the sum of pixel values a and bincludes variance of shot noise of a+b, and similarly to this pixelvalue addition, a pixel value a−b being the difference between the pixelvalues a and b also includes the variance of shot noise of a+b. That is,the noise is added together even at the time of pixel value subtraction,e.g., even with a pixel value of a−a=0 (i.e., black), noise includedtherein is a+a=2a, and this thus causes strong noise in any darkportion.

With the technologies of Patent Documents 2 and 3, no consideration isgiven to dark current noise.

It is thus desirable to easily capture an object image with less noise,balanced shading, and enhanced appearance without the use of a pluralityof lighting devices.

According to an embodiment of the present technology, there is providedan image processing apparatus that includes an ambient-light imageobtaining section, a cumulative image generation section, and ahigh-quality image generation section. The ambient-light image obtainingsection is configured to obtain an ambient-light image in a first timerange, the ambient-light image being an image of an object captured witha predetermined exposure time. The cumulative image generation sectionis configured to generate a cumulative image in a second time rangeafter the first time range, the cumulative image being obtained bycumulative addition of each pixel value in a plurality of images, theplurality of images being of the object captured one by one with thepredetermined exposure time. The high-quality image generation sectionis configured to generate a high-quality image, the high-quality imagebeing obtained by subtracting a pixel value in the ambient-light imagefrom a corresponding pixel value in a normalized image, the normalizedimage being the cumulative image normalized based on a total sum of theexposure time.

The image processing apparatus may further include a light turning-offdetection section that is configured to determine whether or not alighting device is turned off, the lighting device being a light sourcedifferent from a light source from which light is initially emitted forillumination of the object. When the lighting device is determined asbeing turned off, the ambient-light image may be captured.

The lighting device may be turned on during imaging in a time rangeafter the ambient-light image is captured.

The lighting device may be held by a user, and be moved in an arc.

The cumulative image generation section may be configured to perform thecumulative addition of the images of the object captured in the secondtime range, the cumulative addition being performed by classifying theimages by direction based on information specifying toward whichdirections the lighting device emits light. The high-quality imagegeneration section may be configured to generate another high-qualityimage by combining the high-quality images at a predetermined ratio, thehigh-quality images each being obtained by subtracting the pixel valuein the ambient-light image from the corresponding pixel value in thenormalized image, the normalized image being each of the cumulativeimages classified by direction and normalized based on the total sum ofthe exposure time.

The image processing apparatus may further include a display sectionthat is configured to produce an image display. The display section maybe configured to display a GUI that is for specifying the ratio ofcombining the plurality of high-quality images.

The cumulative image generation section may be configured to divide thesecond time range into a plurality of short time ranges, and may beconfigured to perform the cumulative addition of the images of theobject captured in the second time range by classifying the images bythe short time range. The high-quality image generation section may beconfigured to generate another high-quality image by combining thehigh-quality images at a predetermined ratio, the high-quality imageseach being obtained by subtracting the pixel value in the ambient-lightimage from the corresponding pixel value in the normalized image, thenormalized image being each of the cumulative images classified by theshort time range and normalized based on the total sum of the exposuretime.

The image processing apparatus may further include a display sectionthat is configured to produce an image display. The display section maybe configured to display a GUI that is for specifying the ratio ofcombining the plurality of high-quality images.

The image processing apparatus may further include a display sectionthat is configured to produce an image display. In the second timerange, the high-quality image may be sequentially displayed on thedisplay section.

A gain may show a gradual increase before a lapse of a predeterminedtime in the second time range, the gain being multiplied to a luminancevalue of a pixel in the high-quality image displayed on the displaysection.

A gain may show a gradual increase before a maximum luminance value of apixel in the cumulative image reaches a predetermined value, the gainbeing multiplied to a luminance value of a pixel in the high-qualityimage displayed on the display section.

In the cumulative image, a weight coefficient may be multiplied to eachof the pixel values in the plurality of images to prevent a per-imageproportional contribution of the images from being lower than apredetermined value, the images being captured in the second time range.

Occurrence of specular reflection on a surface of the object may bedetected based on a change of a pixel value in the cumulative image, andthe weight coefficient may be changed in value to reduce theproportional contribution of the image observed with the specularreflection.

By a predetermined computing process performed on the pixel values, alighting color may be changed to illuminate the object in the imagescaptured in the second time range.

According to an embodiment of the present technology, there is providedan image processing method that includes obtaining, by an ambient-lightimage obtaining section, an ambient-light image in a first time range,the ambient-light image being an image of an object captured with apredetermined exposure time, generating, by a cumulative imagegeneration section, a cumulative image in a second time range after thefirst time range, the cumulative image being obtained by cumulativeaddition of each pixel value in a plurality of images, the plurality ofimages being of the object captured one by one with the predeterminedexposure time, and generating, by a high-quality image generationsection, a high-quality image, the high-quality image being obtained bysubtracting a pixel value in the ambient-light image from acorresponding pixel value in a normalized image, the normalized imagebeing the cumulative image normalized based on a total sum of theexposure time.

According to an embodiment of the present technology, there is provideda program causing a computer to function as an image processingapparatus that includes an ambient-light image obtaining section, acumulative image generation section, and a high-quality image generationsection. The ambient-light image obtaining section is configured toobtain an ambient-light image in a first time range, the ambient-lightimage being an image of an object captured with a predetermined exposuretime. The cumulative image generation section is configured to generatea cumulative image in a second time range after the first time range,the cumulative image being obtained by cumulative addition of each pixelvalue in a plurality of images, the plurality of images being of theobject captured one by one with the predetermined exposure time. Thehigh-quality image generation section is configured to generate ahigh-quality image, the high-quality image being obtained by subtractinga pixel value in the ambient-light image from a corresponding pixelvalue in a normalized image, the normalized image being the cumulativeimage normalized based on a total sum of the exposure time.

According to a first embodiment of the present technology, in a firsttime range, obtained is an ambient-light image being an image of anobject captured with a predetermined exposure time. In a second timerange after the first time range, generated is a cumulative imageobtained by cumulative addition of each pixel value in a plurality ofimages, the plurality of images being of the object captured one by onewith the predetermined exposure time. A high-quality image is generatedby subtracting a pixel value in the ambient-light image from acorresponding pixel value in a normalized image, the normalized imagebeing the cumulative image normalized based on a total sum of theexposure time.

According to the embodiments of the present technology, an object imageis captured easily with less noise, balanced shading, and enhancedappearance without the use of a plurality of lighting devices.

These and other objects, features and advantages of the presentdisclosure will become more apparent in light of the following detaileddescription of best mode embodiments thereof, as illustrated in theaccompanying drawings.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is a diagram showing an exemplary configuration of an imagingsystem according to an embodiment of the present technology;

FIGS. 2A and 2B are each a diagram illustrating the outer appearance ofa movable lighting device of FIG. 1;

FIG. 3 is a block diagram showing an exemplary internal configuration ofa camera of FIG. 1, and that of the movable lighting device thereof;

FIG. 4 is a block diagram showing an exemplary functional configurationof software such as program to be run by an application processor ofFIG. 3;

FIG. 5 is a diagram illustrating in detail a process by software such asprogram to be run by the application processor;

FIG. 6 is a diagram illustrating in detail a process by software such asprogram to be run by the application processor;

FIG. 7 is a diagram illustrating an exemplary recording format for imagedata of a high-quality image;

FIG. 8 is a flowchart illustrating an exemplary imaging process;

FIG. 9 is a flowchart illustrating an exemplary high-quality imagegeneration process;

FIG. 10 is a diagram illustrating a change of a gain value in responseto a change of cumulative time;

FIG. 11 is a diagram illustrating a change of the gain value in responseto a change of maximum luminance;

FIG. 12 is a diagram illustrating how a pixel value changes duringimaging;

FIG. 13 is a diagram illustrating a mode for rough classification oftypes of shading;

FIG. 14 is a diagram illustrating an example of how a high-quality imageis generated with adjusted shading;

FIG. 15 is a diagram showing an exemplary GUI for adjusting a weightcoefficient;

FIG. 16 is a diagram illustrating another example of how a high-qualityimage is generated with adjusted shading;

FIG. 17 is a diagram showing another exemplary GUI for adjusting theweight coefficient;

FIG. 18 is a diagram showing still another exemplary GUI for adjustingthe weight coefficient; and

FIG. 19 is a block diagram showing an exemplary configuration of apersonal computer.

DETAILED DESCRIPTION OF EMBODIMENT

Hereinafter, an embodiment of the present technology will be describedwith reference to the drawings.

FIG. 1 is a diagram showing an exemplary configuration of an imagingsystem according to the embodiment of the present technology. In FIG. 1,an imaging system 10 is configured to include a movable lighting device11, an object 12, a camera 13, a tripod 15, and an ambient light source16. The camera 13 is provided with a display 14.

The imaging system 10 of FIG. 1 is so configured as to easily capture agood image with better appearance like an image captured using the multilighting system, even by an amateur user.

The movable lighting device 11 is easily moved by a user to any desiredposition. In this example, the rod-shaped movable lighting device 11 isheld by the user, and is moved in an arc as indicated by a dotted arrowin the drawing. In such a manner, the user may illuminate the object 12from his/her desired position only by holding the movable lightingdevice 11 and moving it in an arc. Note that although the movablelighting device 11 in the rod shape is easy to handle, the rod shape isnot restrictive and any other shape is possible.

The object 12 is a target for imaging by the imaging system 10.

The camera 13 is a digital camera that is capable of imaging the object12, and successively obtaining a plurality of images. The camera 13includes therein a camera module, and images the object 12 using thecamera module. This camera module is a device that obtains images with apreset exposure time, and after a lapse of the exposure time, outputsimage data of the object 12 successively to the outside, for example.The detailed configuration of the camera 13 will be described later.

Note that, in this example, although the camera 13 is fixed by thetripod 15, the camera 13 may image the object 12 not using the tripod15.

The display 14 is a display device that presents the image captured bythe camera 13 to the user after execution of a process on the image thatwill be described later. The display 14 is attached to the rear surfaceor the like of the camera 13, and produces a real-time display of imagesto show the time, for example.

Although the details will be described later, the user may end imagingat the point in time when the user thinks the shading is desirable whileviewing the images displayed on the display 14, for example. Moreover,the display 14 is configured by superposition of a touch device, forexample. The display 14 displays thereon images such as GUI (GraphicalUser Interface), and accepts an operation input made by the user basedon the images.

The ambient light source 16 is presumably a lighting device placed in aroom where the object 12 is. The ambient light source 16 is assumed asbeing a general light source such as fluorescent lamp, or an arbitrarylight source that constantly emits light, for example. In the drawing,the ambient light source 16 is shown as a piece of lighting device, butmay be configured by a plurality of lighting devices. If any lightinevitably coming from outside enters the room, this light is consideredalso as being emitted from the ambient light source 16.

That is, the ambient light source 16 collectively means light sourcesother than the movable lighting device 11 that emit light toward theobject 12.

FIGS. 2A and 2B are each a diagram illustrating the outer appearance ofthe movable lighting device 11 of FIG. 1. FIG. 2A shows the rear planeof the movable lighting device 11, and FIG. 2B shows the light-emittingplane (surface) of the movable lighting device 11.

The movable lighting device 11 is provided on the surface with alight-emitting window 22, which is a planar area illuminated by thelight coming from an internal light source.

The movable lighting device 11 is provided on the surface with buttons23-1 to 23-3, which are operated by the user. In this example, providedare the three buttons 23-1 to 23-3, but the number of the buttons is notrestrictive to three. When any of the buttons is depressed, a signalcorresponding to the operation assigned to the depressed button isoutput.

Note that when there is no specific reason to identify these buttons23-1 to 23-3, these buttons are simply referred to as buttons 23.

The movable lighting device 11 except the light-emitting window 22 andthe buttons 23 is covered by an outer shell, which is the shaded area inthe drawing. This outer shell is configured by a low-reflectivitymember. This accordingly protects the object 12 from being illuminatedby reflection of any unwanted light when the user moves the movablelighting device 11 in an arc, for example.

As will be described later, the movable lighting device 11 is providedwith an acceleration sensor, for example, to detect an accelerationvector in six directions that are positive and negative directions ofX-axis, Y-axis, and Z-axis, respectively.

FIG. 3 is a block diagram showing an exemplary internal configuration ofthe camera 13 of FIG. 1, and that of the movable lighting device 11thereof.

In the example of FIG. 3, the camera 13 is configured to include anapplication processor 31, an NVRAM (NonVolatile Random Access Memory)32, a DRAM (Dynamic Random Access Memory) 33, a removable medium 34, adisplay device 35, and a camera module 50. The movable lighting device11 includes a light-emitting device 41, a button 42, and an accelerationsensor 43.

The application processor 31 of FIG. 3 is a core device in a mobiledevice packed with a CPU (Central Processing Unit), a GPU (GraphicsProcessing Unit), a peripheral interface, and the like. The applicationprocessor 31 runs software including a program for controlling imagingby the camera 13, image processing, and the like.

The application processor 31 is connected with various types ofperipheral devices for loading software and data stored in the NVRAM 32to the DRAM 33, for example.

The camera module 50 is a device that obtains images of the object 12 bylight exposure for a preset time, and after the light exposure,successively outputs data of the images to the outside.

As to the image data obtained by capturing an optical image 51 of theobject 12, the camera module 50 is so configured as to transmit theimage data to the application processor 31.

The display device 35 is a functional block that controls image displayon the display 14 of FIG. 1. As described above, the display 14 isequipped with the touch device (not shown), and via this display device,an operation signal is provided to the application processor 31. Thisoperation signal is output in response to the user's touch by finger onthe surface of the display 14.

As to the image data provided after the process by the applicationprocessor 31, the removable medium 34 stores the image data in the formof a file by a recording process, for example. Alternatively, theremovable medium 34 may store software including a program to beinstalled to the camera 13, for example.

The light-emitting device 41 is a light emitter (light source)configured by an LED (Light Emitting Diode), for example. As indicatedby an arrow in the drawing, the application processor 31 is providedwith information as appropriate about whether or not the light-emittingdevice 41 is emitting light.

The button 42 is a functional block of the buttons 23 of FIG. 2B, andmay output a predetermined signal in response to depression by the user.As indicated by an arrow in the drawing, the signal output from thebutton 42 is provided to the application processor 31 as appropriate.

The acceleration sensor 43 detects the acceleration, e.g., detects anacceleration vector in six directions that are positive and negativedirections of X-axis, Y-axis, and Z-axis, respectively. As indicated byan arrow in the drawing, the acceleration detected by the accelerationsensor 43 is provided to the application processor 31 as appropriate.

In this example, the acceleration sensor 43 is provided, but as analternative to the acceleration sensor 43, an angular velocity sensor ora geomagnetic sensor may be provided, for example.

In FIG. 3, arrows between the camera 13 and the movable lighting device11 may be implemented by wireless or wired communication, for example.Alternatively, information about the movable lighting device 11 may beprovided to the application processor 31 in response to a user inputthereof made by operating the camera 13, the touch device of the display14, and the like.

The movable lighting device 11 may be provided with a micro controllerto have the light-emitting device 41, the button 42, and theacceleration sensor 43 operated as peripheral devices.

FIG. 4 is a block diagram showing an exemplary functional configurationof software including a program to be run by the application processor31 of FIG. 3.

In FIG. 4, an individual image obtaining section 71 controls the cameramodule 50 to have it captured a plurality of images of the object 12,thereby obtaining image data of the images. The individual imageobtaining section 71 of FIG. 4 obtains a frame image in each of theimages of the object 12 captured as moving pictures.

Herein, the imaging cycle of the camera module 50 is desirably set to 10frames per second or more to allow the user's interactive adjustment ofshading, for example. Moreover, the exposure time of the camera module50 is desirably set not to cause pixel value saturation even if theobject 12 is illuminated by the movable lighting device 11.

The images obtained by the individual image obtaining section 71 includeimages captured with the movable lighting device 11 turned on, and thenumber of images obtained by the individual image obtaining section 71is determined based on the user operation, for example. Note here thatthe individual image obtaining section 71 may be used also when anambient-light image obtaining section 78 obtains ambient-light images.

In each of the images obtained by the individual image obtaining section71, an addition section 72 performs a pixel value addition for everycorresponding pixel. Herein, the addition section 72 may be used alsowhen the ambient-light image obtaining section 78 obtains theambient-light images.

As to the pixel values obtained for the images by the addition section72, a normalization section 73 normalizes the images by dividing each ofthe pixel values by the number of images subjected to the pixel valueaddition (to be precise, the total sum of the exposure time). Herein,the normalization section 73 may be used also when the ambient-lightimage obtaining section 78 obtains the ambient-light images.

From the pixel values in each of the images after the process ofnormalization by the normalization section 73, a subtraction section 74subtracts pixel values of any corresponding pixels in an image obtainedby the ambient-light image obtaining section 78 that will be describedlater. With this subtraction, the images after the normalization may befree from reflection, shading, and the like resulted from the ambientlight.

A display section 75 provides the display device 35 with display data ofthe images after the process by the subtraction section 74, therebyhaving the display 14 displayed the images after the process by thesubtraction section 74.

A lighting control section 76 controls the movable lighting device 11 tobe turned on or off. The lighting control section 76 controls themovable lighting device 11 to be turned on only when the movablelighting device 11 is determined as moving based on the accelerationdetected by the acceleration sensor 43, for example.

This lighting control section 76 is not necessarily provided, or may beprovided inside of the movable lighting device 11. Moreover, thelighting control section 76 may produce a sound output or a screendisplay to instruct the user to turn off the movable lighting device 11,or to permit the user to turn on the movable lighting device 11.

A light turning-off detection section 77 detects turning-off of themovable lighting device 11. This detection about turning-off of themovable lighting device 11 may be performed by communication, or by auser input made by operating the touch device for his/her confirmationabout turning-off of the movable lighting device 11.

The ambient-light image obtaining section 78 controls the camera module50 to have it captured images of the object 12 with the movable lightingdevice 11 being turned off. The ambient-light image obtaining section 78then obtains image data of the images.

The images obtained by the ambient-light image obtaining section 78 arethose captured with the movable lighting device 11 being turned off,i.e., only those captured in the state that the movable lighting device11 is detected as being turned off by the light turning-off detectionsection 77. The number of images to be obtained by the ambient-lightimage obtaining section 78 may be a piece, or a plurality of images maybe obtained using the individual image obtaining section 71, theaddition section 72, and the normalization section 73, and eventually apiece of ambient-light image may be obtained.

A recording section 79 controls recording of the image data onto theremovable medium 34.

By referring to FIGS. 5 and 6, described in detail is a process bysoftware including a program to be run by the application processor 31.

As described above, the individual image obtaining section 71 controlsthe camera module 50 to have it captured a plurality of images of theobject 12, thereby obtaining image data of the images. In FIG. 5, theleft side shows individual images 101-1 to 101-5, which are images ofthe image data obtained by the individual image obtaining section 71.

In this example, an object for imaging is a spherical substance, andabove the spherical substance, the rod-shaped light-emitting window 22of the movable lighting device 11 is displayed. The individual images101-1 to 101-5 are those captured respectively at times T=1 to T=5.During the imaging, the movable lighting device 11 is moved from left toright in the drawing but not the object.

In this example, the individual images 101-1 to 101-5 are all capturedwhen the movable lighting device 11 is turned on. Alternatively, theuser may turn on the movable lighting device 11 as appropriate. Forexample, the movable lighting device 11 may be turned on for imaging attimes T=1, T=3, and T=4, and the movable lighting device 11 may beturned off for imaging at times T=2, and T=4.

As described above, the addition section 72 performs a pixel valueaddition for every corresponding pixel in each of the images obtained bythe individual image obtaining section 71. The resulting images afterthe pixel value addition are stored as cumulative images in a memorybeing a predetermined storage area in the DRAM 33, for example. In FIG.5, the right side shows cumulative images 102-0 to 102-5.

The cumulative image 102-0 corresponds to an initial value of data inthe memory, and in this example, is an image being entirely black. Theaddition section 72 adds together the cumulative image 102-0 and theindividual image 101-1, thereby generating the cumulative image 102-1.The addition section 72 also adds together the cumulative image 102-1and the individual image 101-2, thereby generating the cumulative image102-2.

With such an addition repeatedly performed, the cumulative image 102-5is generated. In this cumulative image 102-5, the spherical substancebeing the object is brightly displayed as a result of the repeatedaddition of pixel values, and the originally rod-shaped light-emittingwindow 22 of the movable lighting device 11 is displayed like a wideplane.

In such a manner, by cumulatively adding the individual images capturedat various different times, even when the lighting device in use is onlythe movable lighting device 11, the virtual lighting effect may beobtained as if a plurality of lighting devices were spatially disposed.Therefore, according to the embodiment of the present technology, evenwhen a user is amateur with no expertise, images to be captured therebyhave enhanced appearance with ease similarly to those captured by usingthe multi lighting system.

In this example, for brevity, exemplified is the case of cumulativelyadding the individual images captured at times T=1 to T=5. However,actually, several tens to hundreds of individual images are captured andcumulatively added together. For the user's interactive adjustment ofshading, the screen of the display 14 is updated desirably for every tenframes per second or more, and desirably with a frequency similarthereto, the individual images are to be captured.

Also for the user's interactive adjustment of shading, the movablelighting device 11 is expected to be held and moved in an arc by theuser for several to a dozen or so seconds to create shading. If this isthe case, the number of the individual images is several tens tohundreds in total.

When a large number of individual images are cumulatively addedtogether, noise components such as shot noise superposed on theindividual images at all times are significantly reduced so that theresulting images are improved in quality. Further, cumulatively addingtogether several hundreds of individual images produces the great noisereduction effect, for example. This thus leads to images in which theeffect of noise such as light shot noise is significantly reducedwithout using a lighting device that emits strong light. That is,according to the embodiment of the present technology, the noisereduction effect is secondarily produced.

Still further, by reducing the exposure time at the same time asreducing the imaging cycle, imaging is prevented from failing due topixel value saturation that often occurs during imaging with multipleexposure, for example.

As described above, as to the pixel values obtained in the images by theaddition section 72, the normalization section 73 normalizes the imagesby dividing each of the pixel values by the number of images subjectedto the pixel value addition. By taking the cumulative image 102-5 as anexample, this image is the result of cumulatively adding five individualimages, and thus is normalized by dividing each pixel value by 5.

Note that, actually, the exposure time is not fixed at all times duringimaging of the individual images 101-1 to 101-5. In considerationthereof, each pixel value in the cumulative image 102-5 is divideddesirably by the total exposure time spent for imaging of the individualimages 101-1 to 101-5. Hereinafter, the total exposure time is referredto as cumulative time.

In the example of FIG. 6, the cumulative image 102-5 is normalized bydividing each pixel value therein by the cumulative time.

As described above, the ambient-light image obtaining section 78controls the camera module 50 to have it captured images of the object12 in the state that the movable lighting device 11 is turned off, andobtains image data of the images. The example of FIG. 6 shows anambient-light image 110 corresponding to image data obtained by theambient-light image obtaining section 78.

The ambient-light image 110 is the image captured when the movablelighting device 11 is turned off, and the example of FIG. 6 shows theimage of a dimly-lit object. The ambient-light image 110 may be an imagecaptured at a specific time, or by cumulatively adding images capturedat various different times.

For generating the ambient-light image 110 by cumulatively adding imagescaptured at various different times, similarly to the case describedabove by referring to FIG. 5, the addition section 72 performs pixelvalue addition, and the normalization section 73 performs normalization.This accordingly produces the noise reduction effect also in theambient-light image 110. That is, with normalization performed bycumulatively adding a plurality of images, an arbitrary number ofambient-light images may be cumulatively added together until the shotnoise (including noise floor) is reduced to a sufficient level.

When the ambient-light image 110 is an image captured at a specifictime, noise reduction is desirably accomplished by a very-effectivenoise removal technique such as bilateral filtering or NL-MEANSfiltering.

In the example of FIG. 6, the ambient-light image 110 is the onegenerated by cumulatively adding images captured at various differenttimes, and each pixel value in the ambient-light image 110 is divided bythe cumulative time.

Moreover, as described above, from the pixel values in each of theimages after the process of normalization by the normalization section73, the subtraction section 74 subtracts pixel values of anycorresponding pixels in the image obtained by the ambient-light imageobtaining section 78. That is, because the cumulative image includesambient-light components at all times, by removing the ambient-lightcomponents, the resulting image has shading as if it were captured in adarkroom.

In the example of FIG. 6, from the pixel values in the normalized imageof the cumulative image 102-5, the pixel values in the normalized imageof the ambient-light image 110 are subtracted so that a high-qualityimage 111-5 is generated. As shown in the drawing, the high-qualityimage 111-5 does not have the shadow observed in the ambient-light image110, i.e., on the left side of the spherical substance being the object.

In this manner, the resulting image may have shading the same as that ofan object image captured using the multi lighting system in a darkroomwith no ambient light, for example.

Also in this manner, the shot noise in the ambient-light components isreduced, thereby being able to cancel out the dark current noiseincluded in the ambient-light noise and that in the light componentsfrom the movable lighting device.

The high-quality image 111-5 is then presented to the user by beingdisplayed on the display 14 by the display section 75. Note that thehigh-quality image is updated every time an individual image isobtained. Therefore, similarly to a case of displaying images of movingpictures, while the user images an object, the updated image iscontinuously displayed on the display. The user moves the movablelighting device 11 while looking at the high-quality image on thedisplay 14, and when the user thinks that the high-quality image hashis/her desired shading, the user issues a command to end the imaging.

As described above, the recording section 79 controls recording of theimage data of the high-quality image onto the removable medium 34. Atthe time of recording of the image data of the high-quality image, thedata is recorded in the recording format as shown in FIG. 7.

In the example of FIG. 7, a recording unit of a file 141 includes imagedata 143, and meta data 142. In this example, the meta data 142 isconfigured by three tags of height, width, and type. The tag of type inthe meta data is information for use to determine whether or not theimage is a high-quality image according to the embodiment of the presenttechnology.

The use of such tags allows easy separation only of high-quality images,and leads to easy image search and display, for example.

In such a manner, according to the embodiment of the present technology,images are obtained with the similar quality as those captured byfull-scale imaging with the use of multi lighting system in a darkroom.

Described next is an exemplary imaging process by the imaging system 10by referring to the flowchart of FIG. 8.

In step S21, the lighting control section 76 turns off the movablelighting device 11. At this time, the movable light device 11 may beturned off by wireless or wired signal transmission, or a message may bedisplayed on the display 14 to instruct turning-off of the movablelighting device 11, for example.

In step S22, the light turning-off detection section 77 determineswhether or not turning-off of the movable lighting device 11 isdetected. This detection of turning-off of the movable lighting device11 may be performed by wireless or wired communication, or the user maymake an input by operating the touch device for his/her confirmationabout turning-off of the device, for example.

In step S22, when a determination is made that the turning-off of thelight is not detected, the procedure returns to step S21. At this time,an error message may be displayed on the display 14, or a warning may beissued by sound to prompt turning-off of the light, for example.

On the other hand, when a determination is made that the turning-off ofthe device is detected, the procedure goes to step S23.

In step S23, the ambient-light image obtaining section 78 controls thecamera module 50 to have it captured an image of the object 12, andobtains image data of the image.

In step S24, the lighting control section 76 permits turning-on of themovable lighting device 11. At this time, the movable lighting device 11may be turned on by wireless or wired signal transmission, or a messagemay be displayed on the display 14 to permit turning-on of the movablelighting device 11, for example.

In step S25, a high-quality image generation process is performed, whichwill be described later by referring to the flowchart of FIG. 9. Withthis process, a high-quality image is generated.

In step S26, the display section 75 provides the display device 35 withdisplay data of the image, which is obtained by the process of thesubtraction section 74. The display section 75 then has the display 14displayed the image obtained by the process of the subtraction section74.

In step S27, a determination is made whether or not a command is issuedto end the imaging. This command may be issued by depression of a buttonor the like of the camera 13, or may be issued by depression of any ofthe buttons 23 of the movable lighting device 11, for example.Alternatively, the command may be issued automatically to end theimaging when the high-quality image generated in step S25 is detected asremaining almost the same.

Still alternatively, the high-quality image generation process may betemporarily stopped when detection is made that the user stopsilluminating the object by the movable lighting image 11. This mayprevent cumulative addition of images that are captured in the durationbetween the stopping of object illumination and the depression of thebutton or the like of the camera 13.

Herein, the detection whether or not the user stops illuminating theobject may be made by detecting turning off of the light based on theoperation of the buttons 23 of the movable lighting device, or when apixel value change in the high-quality image becomes smaller than apredetermined value, for example. Still alternatively, image recognitionmay be used as a basis to detect turning-off of light. When a pixelvalue difference between the individual image and the ambient-lightimage falls within a predetermined range, detection may be made that thelight is turned off. Still alternatively, sound detection may be used asa basis to detect that the user stops illuminating the object. When thelevel of the sound becomes higher than a predetermined value, or whenpredetermined sound is acknowledged, detection may be made that the userstops illuminating the object.

When a preset given trigger is detected, for example, the high-qualityimage generation process is resumed. This trigger may be issued whenturning-on of the light is detected based on the operation of thebuttons 23, or when turning-on of the light is detected by imagerecognition (when a pixel value difference between the captured imageand the ambient-light image becomes a predetermined value or larger),for example. Still alternatively, the trigger may be issued when anysound is detected, e.g., when the level of the sound becomes higher thana predetermined value, or when a predetermined sound is recognized.

In step S27, when a determination is made that the command is not yetissued to end the imaging, the procedure returns to step S25, and theprocesses thereafter are repeated. On the other hand, in step S27, whena determination is made that the command is issued to end the imaging,the procedure goes to step S28.

In step S28, the recording section 79 generates a file in thepredetermined recording format including the image data of thehigh-quality image generated in the process of step S25. At this time,generated is the file described above by referring to FIG. 7, forexample.

Then in step S29, the file generated in step S28 is recorded. In thismanner, the imaging process is performed.

Next, by referring to the flowchart of FIG. 9, described is a detailedexample of the high-quality image generation process in step S25 of FIG.8.

In step S41, the individual image obtaining section 71 obtainsindividual images. At this time, obtained are the individual images101-1 to 101-5 of FIG. 5 one by one.

In step S42, the addition section 72 cumulatively adds the individualimages obtained by the individual image obtaining section 71. At thistime, as described above by referring to FIG. 5, each of the individualimages is added with the cumulative image one generation before, therebysuccessively generating the cumulative images 102-1 to 102-5, forexample.

In step S43, the normalization section 73 normalizes each of thecumulative images obtained by the process of step S42 by dividing thepixel value by the cumulative time. As shown in FIG. 6, for example,each pixel value in the cumulative images 102-5 is divided by thecumulative time.

When generating an ambient-light image by cumulatively adding imagescaptured at various different times, the processes from step S41 to S43are also performed.

In step S44, from the pixel values in each of the images obtained by theprocess of step S43, the subtraction section 74 subtracts the pixelvalues of corresponding pixels in the corresponding ambient-light image.At this time, as shown in FIG. 6, from the pixel values in thenormalized image of the cumulative image 102-5, the pixel values ofcorresponding pixels in the ambient-light image 110 are subtracted sothat the high-quality image 111-5 is generated, for example.

In this manner, the high-quality image is generated. Herein, exemplifiedis the case that the process of obtaining an ambient-light image in stepS23 is performed first, and then processes related to generating ahigh-quality image in steps S25 to S27 are performed. This is notrestrictive, and the processes may be performed in any different order,e.g., processes of steps S25 to S27 may be performed first to generate ahigh-quality image by estimating an ambient-light image somehow, or bygenerating a temporary ambient-light image. If this is the case, ahigh-quality image may be obtained by imaging an ambient-light imageafter performing the processes in steps to generate a high-qualityimage, and then replacing the ambient-light image with the temporaryambient-light image. Alternatively, by alternately repeating incrementalprocesses of step S23 (obtaining an ambient-light image) and steps S25to S27 (generating a high-quality image) for several times in the samestate, a high-quality image may be obtained while updating a temporaryambient-light image in a short cycle.

With the so-called bulb imaging being a part of the pervious long-timeexposure imaging, first of all, an image exposed to light by an openedmechanical shutter is stored in a memory, and after the light exposure,the mechanical shutter is closed to accumulate the dark currentcomponents. Thereafter, from the image stored in the memory, the darkcurrent components are removed. The effect produced by such long-timeexposure is similarly produced also by imaging a plurality of images oneby one for cumulative addition.

With such bulb imaging, however, the dark-current components areaccumulated with the mechanical shutter closed. This does not remove thecomponents of the ambient-light image as does the embodiment of thepresent technology.

There is also an imaging mode called firework mode, for example, withwhich a plurality of images are captured one by one for cumulativeaddition. Even with such an imaging mode, no consideration is given forremoval of the ambient-light components.

On the other hand, according to the embodiment of the presenttechnology, an ambient-light image is first captured with an openedmechanical shutter, and after this imaging, the mechanical shutterremains open to capture a plurality of individual images for cumulativeaddition. From the resulting cumulative images, the components of theambient-light image are to be removed. Needless to say, imagingaccording to the embodiment of the present technology is possible evenwith an inexpensive camera not provided with a mechanical shutter.

Therefore, according to the embodiment of the present technology, anynoise resulted from the dark-current components generated by long-timeexposure may be prevented from being superposed on images. Moreover,components of ambient-light images causing user's unwanted shading orthe like may be removed.

As described above, cumulative addition of a large number of individualimages significantly reduces components of noise such as shot noisebeing superposed on the individual images at all times. However, whenthe individual images for cumulative addition are small in number, someof the noise components may remain. That is, the high-quality image maynot be free from noise before a lapse of sometime after the start ofimaging.

Accordingly, before the lapse of a predetermined time after the start ofimaging, images may be displayed on the display 14 with reducedluminance or the like not to enhance the noise, for example.

As an example, a vector containing the value of each pixel is used torepresent an image, and the luminance of the pixels is adjusted asindicated by Equation 1.

$\begin{matrix}{{{\overset{->}{I}}^{\prime} = {k \cdot \overset{->}{I}}}\left\{ \begin{matrix}{k = \frac{t}{t_{0}}} & {{{if}\mspace{14mu} t} < t_{0}} \\{k = 1} & {otherwise}\end{matrix} \right.} & (1)\end{matrix}$

In Equation 1, a vector I represents the luminance value of ahigh-quality image provided as an input image, and a vector I′represents the luminance value of a high-quality image displayed on thedisplay 14. Also in Equation 1, a variable t denotes the cumulativetime, and t0 denotes a predetermined constant time. Also in Equation 1,k is a gain for multiplication of the luminance value of the pixels.

The gain k in Equation 1 varies as shown in FIG. 10, for example. InFIG. 10, the horizontal axis indicates the cumulative time t, and thevertical axis indicates the value of the gain k. The value change of thegain k responding to the change of the cumulative time is indicated by aline 151.

As shown in FIG. 10, the value of the gain k shows a gradual increaseuntil the cumulative time reaches t0, and is fixed to 1 after thecumulative time reaches t0.

With the computing process of Equation 1 using the gain k as shown inFIG. 10, images on the display 14 are displayed with reduced luminancenot to enhance the noise before the lapse of a predetermined time afterthe start of imaging.

Alternatively, by paying attention to the maximum value of the luminanceof a high-quality image, the luminance of the pixels may be adjusted asindicated by Equation 2.

$\begin{matrix}{{{\overset{->}{I}}^{\prime} = {k \cdot \overset{->}{I}}}\left\{ \begin{matrix}{k = \frac{\max\left( \overset{->}{Y} \right)}{Y_{0}}} & {{{if}\mspace{14mu}{\max\left( \overset{->}{Y} \right)}} < Y_{0}} \\{k = 1} & {otherwise}\end{matrix} \right.} & (2)\end{matrix}$

In Equation 2, the vector I represents the luminance value of ahigh-quality image provided as an input image, and the vector I′represents the luminance value of a high-quality image displayed on thedisplay 14. Also in Equation 2, a vector Y represents the pixel value ina cumulative image, and max(Y) denotes the maximum pixel value (maximumluminance) in the cumulative image. Also in Equation 2, Y0 denotespredetermined luminance at which the cumulative image becomes brighter,and the shot noise components are determined to be small relative to aluminance signal. Also in Equation 2, k is a gain for multiplication ofthe luminance value of the pixels.

The gain k in Equation 2 varies as shown in FIG. 11, for example. InFIG. 11, the horizontal axis denotes the maximum luminance max(Y), andthe vertical axis denotes the value of the gain k. The value change ofthe gain k responding to the change of the maximum luminance isindicated by a line 152.

As shown in FIG. 11, the value of the gain k shows a gradual increaseuntil the maximum luminance reaches Y0, and is fixed to 1 after themaximum luminance reaches Y0.

With the computing process of Equation 2 using the gain k as shown inFIG. 11, images on the display 14 are displayed with reduced luminancenot to enhance the noise until a predetermined number of images arecumulatively added together.

With the lapse of sufficiently a long time after the start of imaging,if the cumulatively-added individual images are very large in number,the resulting high-quality image looks almost the same. This is becausewhen the cumulatively-added individual images are increased in number,the contribution made by each of the individual images to thehigh-quality image becomes proportionally less and less (hereinafter,this contribution is referred to as proportional contribution).

The addition process by the addition section 72 is represented byEquation 3.

$\begin{matrix}{\begin{bmatrix}{\overset{->}{Y}}_{0} \\t_{0}\end{bmatrix} = {{\begin{bmatrix}\overset{->}{0} \\0\end{bmatrix}\begin{bmatrix}{\overset{->}{Y}}_{k} \\t_{k}\end{bmatrix}} = \begin{bmatrix}{{\overset{->}{Y}}_{k - 1} + {\overset{->}{X}}_{k}} \\{t_{k - 1} + e_{k}}\end{bmatrix}}} & (3)\end{matrix}$

In Equation 3, a vector Yk represents the pixel value in a cumulativeimage of k pieces of cumulatively-added individual images, and a vectorXk represents the pixel value in the k-th individual image. Also inEquation 3, tk denotes the exposure time in total (cumulative time) whenthe k-th individual image is captured, and ek denotes the exposure timeduring imaging of the k-th individual image. A vector Y0 representingthe initial pixel value in the cumulative image is a 0 vector, and thecumulative time t0 at this time is 0.

In order to prevent reduction of the proportional contribution perindividual image to be caused by the increased number ofcumulatively-added individual images, the addition section 72 mayperform the addition process as represented by Equation 4.

$\begin{matrix}{\begin{bmatrix}{\overset{->}{Y}}_{0} \\t_{0}\end{bmatrix} = {{\begin{bmatrix}\overset{->}{0} \\0\end{bmatrix}\begin{bmatrix}{\overset{->}{Y}}_{k} \\t_{k}\end{bmatrix}} = \begin{bmatrix}{{\left( {1 - w} \right) \cdot {\overset{->}{Y}}_{k - 1}} + {w \cdot {\overset{->}{X}}_{k}}} \\{{\left( {1 - w} \right) \cdot t_{k - 1}} + {w \cdot e_{k}}}\end{bmatrix}}} & (4)\end{matrix}$

In Equation 4, w is a parameter representing the weight for adjustingthe proportional contribution of the individual images. The computingprocess by Equation 4 may be a process using a kind of IIR filter.

That is, with Equation 4, by adding the parameter of weight w and (1-w)to Equation 3, the value of w may determine the minimum proportionalcontribution of each of the individual images to the cumulative image.The parameter w may be a preset constant, for example.

When the value of w is appropriately set, even with a lapse ofsufficiently a long time after the start of imaging, any change of thehigh-quality image on the display 14 becomes perceivable at all times.

The light source of the movable lighting device 11 is assumed as beingwhite in color. Desirably, the light source emits light of variousdifferent colors depending on the characteristics of an object, thepurpose of use of images, and the like. In this case, exchanging thelight source of the movable lighting device 11 will do, but if imageprocessing realizes the lighting effect of various colors, the user'sconvenience may be increased to a further degree.

That is, pixel values in an image captured as an individual image areclassified into three elements of R, G, and B. These elements are eachapplied with a matrix filter including positive coefficients of Wr, Wg,and Wb, and the resulting R′, G′, and B′ are used as the pixel values ofthe individual image. This accordingly implements imaging using lightingin color determined by the coefficients of Wr, Wg, and Wr.

That is, image processing with computing as Equation 5 may be performedby the individual image obtaining section 71, for example.

$\begin{matrix}{\begin{bmatrix}R^{\prime} \\G^{\prime} \\B^{\prime}\end{bmatrix} = {\begin{bmatrix}W_{r} & 0 & 0 \\0 & W_{g} & 0 \\0 & 0 & W_{b}\end{bmatrix}\begin{bmatrix}R \\G \\B\end{bmatrix}}} & (5)\end{matrix}$

The lighting effect in various different colors produced by such imageprocessing is implemented by the user operation based on the GUIdisplayed on the display 14, for example. Alternatively, it may beconvenient if a control signal is output to produce the lighting effectin desired color in response to depression of the buttons 23 of themovable lighting device 11, for example.

In the above, Wr, Wg, and Wb are each described as a positivecoefficient. Alternatively, these Wr, Wg, and Wb may be negativecoefficients, and if this is the case, produced is the lighting effectof darkening too-bright color components.

When light reflects on the surface of the substance being an object,specular reflection may occur, i.e., reflection of light components ofthe light source irrespective of the surface color of the object. Ifunwanted specular reflection occurs when the movable lighting device 11is moved to emit light to the object 12, reducing the light componentsto be reflected by specular reflection is desirable.

FIG. 12 is a diagram illustrating how a pixel value shows a changeduring imaging. That is, FIG. 12 illustrates how a color component of apixel at a predetermined position in a high-quality image (or in acumulative image) shows a change by repeated cumulative addition in thespace in which R, G, and B components are each a straight axis.

As an example, the color represented by a vector C1 at time T=1 ischanged to the color represented by a vector C2 at time T=2. In thiscase, with illumination by light, the color is changed to be brighteralong a straight line La that maintains the hue of the surface color ofthe object 12.

On the other hand, the color represented by a vector C3 at time T=3 ischanged to the color represented by a vector C4 at time T=4. That is,from time T3 to T4, the color is changed to be brighter along a straightline Lb, which extends toward (1, 1, 1) representing the white color ofthe light source of the movable lighting device 11.

The reason of the path change from the straight line La to Lb is theoccurrence of specular reflection, i.e., reflection of light componentsof the light source irrespective of the surface color of the object whenlight reflects on the surface of the object 12. Herein, the reflectedlight relevant to the straight line La is diffuse reflection components,and the reflected light relevant to the straight line Lb is specularreflection components.

For reducing such specular reflection components, exemplarily inEquation 4 above, the parameter w may be dynamically changed to reducethe proportional contribution when the specular reflection componentsare included.

To be specific, in the addition section 72, a pixel value analysis isconducted for an individual image, and as to the color components ofpixels in a cumulative image, any change is specified in the space inwhich R, G, and B components are each a straight axis. In this case, thevector as described above by referring to FIG. 12 is obtained for everypixel, for example.

The direction of the straight line La in FIG. 12 may be determined inadvance for every pixel based on an ambient-light image, for example.

As the vectors C1 and C2 in FIG. 12, when a vector representing thepixel color moves along the straight line La that maintains the hue ofthe surface color of the object 12, for example, the addition section 72may use the parameter w of a predetermined value for computation withEquation 4. On the other hand, as the vectors C3 and C4 in FIG. 12, whena vector representing the pixel color moves along the straight line Lbthat extends toward (1, 1, 1) representing the white color of the lightsource, the parameter w may be set to a smaller value in Equation 4.

As an example, for each pixel, a difference of direction between thestraight line La and each of the color vectors C1 to C4 is computed byinner product calculation or the like using the angle as an evaluationvalue. The specular reflection components are then defined by strengthby calculating the evaluation value of every pixel in a cumulativeimage, thereby setting the parameter w to be appropriate to the strengthof the specular reflection components.

In this manner, specular reflection components observed in ahigh-quality image may be reduced, for example.

For actual imaging using the multi lighting system, a plurality oflighting devices are disposed around the object, for example. In thiscase, the lighting devices may vary in lighting intensity to captureimages with characteristic shading. According to the embodiment of thepresent technology, the movable lighting device 11 is solely used, butthe shading in the resulting high-quality image may be similar to thatin an image captured using the multi lighting system in which lightingdevices vary in lighting intensity.

For example, for generating a cumulative image, a weight coefficientappropriate to the state of shading (type of shading) may be used formultiplication at the time of cumulative addition of individual images.This accordingly leads to a high-quality image with shading similar tothat in an image captured using a plurality of lighting devices varyingin lighting intensity.

As the individual images 101-1 to 101-5 of FIG. 5, the images of theobject captured by the camera 13 have different shading depending on theposition of the movable lighting device 11. This means a larger numberof individual images lead to a larger number of types of shading,thereby resulting in a difficulty in finding a desired type of shading.Therefore, in order to use a weight coefficient appropriate to the typeof shading for multiplication at the time of generating a cumulativeimage, the types of shading are expected to be roughly classified inadvance.

FIG. 13 is a diagram illustrating a mode for rough classification oftypes of shading. In this example, the types of shading are classifiedon the assumption that there is a correlation between the orientation ofthe light-emitting window 22 of the movable lighting device 11 and theshading of the object 12.

As described above, the movable lighting device 11 is provided with theacceleration sensor 43, and the acceleration sensor 43 detects theacceleration, e.g., detects an acceleration vector in six directionsthat are positive and negative directions of X-axis, Y-axis, and Z-axis,respectively, in FIGS. 2A and 2B.

FIG. 13 shows unit vectors (vectors Va, Vb, Vc, Vd, Ve, and Vf) of sixvectors extending along the X, Y, and Z axes from the origin in thelocal coordinates of the movable lighting device 11, and a unit vector(vector S) representing the direction of the object 12.

For detecting the unit vector S representing the direction toward theobject 12, for example, the acceleration sensor 43 is used to detect thedirection of the gravity. The direction of the gravity is then mapped onthe local coordinates of the movable lighting device 11, and thedirection of gravity is used as the direction toward the object 12.

The manner of detecting the unit vector S representing the directiontoward the object 12 is not restrictive to the above, and any othersensors, e.g., gyro sensor and geomagnetic sensor, may be used together.Alternatively, a camera attached to the movable lighting device 11 maybe used to specify the position of the object, or a three-dimensionalposition sensor may be used.

The inner product between the vectors is defined as below.

That is, the inner product Pa is defined as Pa=(Va, S), and the innerproduct Pb is defined as Pb=(Vb, S). Similarly, the inner products Pc toPf are respectively defined as Pc=(Vc, S), Pd=(Vd, S), Pe=(Ve, S), andPf=(Vf, S). Herein, with a comparison of value performed among the innerproducts Pa to Pf, finding the vector with the maximum inner productleads to identifying the vector closest in direction toward the objectamong the six vectors extending along the X-axis, Y-axis, and Z-axis.The vector closest in direction toward the object is referred to asvector Vx. That is, the vector Vx is any of the vectors Va, Vb, Vc, Vd,Ve, and Vf.

When the addition section 72 cumulatively adds together the individualimages, the image data of the cumulative images may be stored indifferent memories provided for each of the above six vectors, i.e.,vectors Va, Vb, Vc, Vd, Ve, and Vf. Thereafter, any of the memoriescorresponding to the vector Vx is selected for cumulative addition, andusing the cumulative images stored in each of the memories, sixhigh-quality images are generated.

Moreover, the six high-quality images generated as described above maybe combined together at an arbitrary ratio, thereby generating ahigh-quality image with adjusted shading.

FIG. 14 is a diagram illustrating an example of how to generate ahigh-quality image with adjusted shading as described above. In FIG. 14,images 111Va to 111Vf are six high-quality images generated based on thecumulative images stored in different memories provided for each of thevectors as described above.

As shown in FIG. 14, pixels in the images 111Va to 111Vf are eachmultiplied by the weight coefficient (Wa, Wb, Wc, Wd, We, or Wf) beforecumulative addition so that a high-quality image 111 is obtained withadjusted shading. Herein, the weight coefficients may be so set as tosatisfy the condition of Wa+Wb+Wc+Wd+We+Wf=1.

Alternatively, the high-quality image 111 with adjusted shading may beobtained by adjusting the weight coefficients based on the useroperation on the images 111Va to 111Vf displayed on the display 14.

FIG. 15 is a diagram showing an exemplary GUI for use to adjust theweight coefficients Wa to Wf. This GUI is displayed on the display 14during or after imaging of the object, for example.

In FIG. 15, an area 161 is for display of the six high-quality imagesgenerated based on the cumulative images stored in different memoriesprovided for each of the vectors. As shown in the drawing, in the area161, the images 111Va to 111Vf are displayed.

In FIG. 15, an area 162 is for display of a circle graph indicating theratio of the respective weight coefficients Wa to Wf. The circle graphin the drawing shows Va, Vb, Vc, and the like, each of which representsthe ratio of the weight coefficient Wa, Wb, Wc, or the like. The user isallowed to adjust the ratio of each of the weight coefficients Wa to Wfby a finger touch on the circle graph displayed in the area 162, forexample.

In FIG. 15, an area 163 is for display of the high-quality image 111with adjusted shading. This high-quality image 111 is obtained bycumulative addition of the images 111Va to 111Vf after multiplication ofthe weight coefficients to the pixels therein. The weight coefficientsare those satisfying the condition of Wa+Wb+We+Wd+We+Wf=1, and being setto the ratio in the circle graph in the area 162.

In such a manner, the types of shading are classified based on theorientation of the light-emitting window 22 of the movable lightingdevice 11, and images appropriate to each type of shading are combinedtogether at the user's preferred ratio so that a high-quality image isobtained. In this manner, the resulting high-quality image may haveshading similar to that in an image captured using the multi lightingsystem in which lighting devices vary in lighting intensity.

Herein, a high-quality image may be obtained with adjusted shading in adifferent manner from above.

As an example, a plurality of high-quality images may be generated basedon the imaging time of individual images, and these high-quality imagesmay be combined at the user's preferred ratio to generate a high-qualityimage. In this case, the types of shading are classified on theassumption that there is a correlation between the imaging time and theshading of the object 12. That is, on the assumption that the individualimages captured in a short time range have almost the same shading, aplurality of high-quality images are generated.

As an example, cumulative images are stored in different memoriesprovided for each predetermined time range, and individual images arecumulatively added together in the time range including the imaging timethereof. For example, provided are memories corresponding to six timeranges, i.e., times T=0 to T=9, times T=10 to T=19, times T=20 to T=29,times T=30 to T=39, times T=40 to T=49, and times T=50 to T=59. Based onthe cumulative images stored in these memories, six high-quality imagesare generated. By combining these high-quality images at a predeterminedratio, a high-quality image is generated with adjusted shading.

FIG. 16 is a diagram illustrating an example of how to generate ahigh-quality image with adjusted shading as described above. In FIG. 16,images 111 (T=0 . . . 9) to 111 (T=50.59) are six high-quality imagesdescribed above. The image 111 (T=0 . . . 9) is a high-quality imagegenerated based on cumulative images being the result of cumulativeaddition of 10 individual images captured in a time range from times T=0to T=9, and the image 111 (T=10.19) is a high-quality image generatedbased on cumulative images being the result of cumulative addition of 10individual images captured in a time range from times T=10 to T=19, forexample.

As shown in FIG. 16, by multiplying a weight coefficient to each of thepixels in the images 111 (T=0 . . . 9) to 111 (T=50 . . . 59), thehigh-quality image 111 with adjusted shading is obtained. The weightcoefficients are W(T=0 . . . 9), W(T=10 . . . 19), W(T=20 . . . 29),W(T=30 . . . 39), W(T=40 . . . 49), and W(T=50 . . . 59). These weightcoefficients may be so set as to satisfy the condition that the totalsum thereof is 1.

Alternatively, the high-quality image 111 may be obtained with adjustedshading by adjusting the weight coefficients by user operation with theimages 111 (T=0 . . . 9) to 111 (T=50 . . . 59) displayed on the display14.

FIG. 17 is a diagram showing an exemplary GUI for use to adjust theweight coefficients of W(T=0 . . . 9) to W(T=50 . . . 59). This GUI isdisplayed on the display 14 during or after imaging of the object, forexample.

In FIG. 17, an area 171 is for display of the high-quality image 111with adjusted shading. Also in FIG. 17, an area 172 is for display ofthe images 111(T=0 . . . 9) to 111 (T=50 . . . 59), and below theseimages 111(T=0 . . . 9) to 111 (T=50 . . . 59), a bar 173 a is displayedtogether with a scale indicating the time T. This bar 173 a is used as abasis to specify which images are to be combined at what ratio.

In the example of FIG. 17, the combination ratio of the image 111 (T=0 .. . 9) (or the weight coefficient W (T=0 . . . 9)) is 0%, thecombination ratio of each of the images 111 (T=10 . . . 19) and 111(T=50 . . . 59) is 12.5%, and the combination ratio of each of theimages 111 (T=20 . . . 29) to 111 (T=40 . . . 49) is 25%.

Alternatively, bars 173 b-1 and 173 b-2 of FIG. 18 may be used as abasis to specify which images are to be combined at what ratio. In theexample of FIG. 18, the combination ratio of each of the images 111(T=10 . . . 19) and 111 (T=50 . . . 59) is 50%, and the combinationratio of each of the remaining images is 0%.

Herein, the bars 173 a and 173 b are each an element of the GUIdisplayed on the display based on the user operation.

In such a manner, by combining high-quality images generated on the timerange basis to obtain a high-quality image with adjusted shading, evenwith any image captured by the user illuminating the object 12 from awrong direction using the movable lighting device 11, the resultingimage may not be affected in terms of shading. That is, even if the userfeels “Oops!” in the process of imaging, the effect caused thereby maybe removed.

For rough classification of the types of shading, using the directionsand times as above is not restrictive, and comparing the degree ofsimilarity of images for classification is also a possibility. Forclassification by direction, using parameters of six degrees of freedomis more desirable, which represent the direction and position of a rigidbody in the space.

In the embodiment described above, image processing may cancel anyunexpected appearance of a part of the movable lighting device or theuser's hand (a part of the user's body) in the angle of view, forexample. To be specific, any portion determined to be a moving object bythe general recognition technique of moving objects may be replaced witha portion extracted from an ambient-light image so that an individualimage may be corrected for cumulative addition. Alternatively, in afinal high-quality image, any portion determined by the recognitiontechnique of moving images to be affected may be replaced with a portionextracted from an ambient-light image.

The series of processes described above may be performed by hardware orsoftware. For the software to perform the series of processes describedabove, a program in the software is installed over a network or from arecording medium on a computer in a hardware specifically designedtherefor, or on such a general-purpose personal computer 700 as shown inFIG. 19 that may perform various functions by installation of variousprograms, for example.

In FIG. 19, a CPU (Central Processing Unit) 701 performs various typesof processes in accordance with a program stored in a ROM (Read OnlyMemory) 702, or a program loaded from a storage section 708 to a RAM(Random Access Memory) 703. The RAM 703 stores data for the CPU 701 toperform various processes as appropriate.

The CPU 701, the ROM 702, and the RAM 703 are connected to one anothervia a bus 704. This bus 704 is connected also with an input/outputinterface 705.

The input/output interface 705 is connected with an input section 706,an output section 707, a storage section 708, and a communication unit709. The input section 706 includes a keyboard, a mouse, and the like,and the output section 707 includes a display exemplified by LCD (LiquidCrystal Display), a speaker, and the like. The storage section 708includes a hard disk, for example, and the communication unit 709includes a modem, a network interface card exemplified by a LAN(Local-Area Network) card, and the like. The communication unit 709performs a communication process over a network including the Internet.

The input/output interface 705 is connected with a drive 710 asappropriate, and a removable medium 711 including a magnetic disk, anoptical disk, a magneto-optical disk, and a semiconductor memory isattached as appropriate. A computer program read from the removablemedium 711 is installed on the storage section 708 as appropriate.

When the series of processes described above are performed by software,programs in the software are installed over a network including theInternet, or from a recording medium including the removable medium 711,for example.

Note here that this recording medium is not restrictive to the removablemedium 711 of FIG. 19 that is provided separately from the device body,and includes program-recorded disks for program distribution to users,e.g., a magnetic disk (including a floppy disk (trademark)), an opticaldisk (including a CD-ROM (Compact Disk-Read Only Memory), a DVD (DigitalVersatile Disk)), and a magneto-optical disk (including an MD(Mini-Disk)™), and a semiconductor memory. The recording medium may alsoinclude the program-recorded ROM 702, a hard disk in the storage section708, and the like that are incorporated in advance in the device bodyfor distribution to the users.

Note that the series of processes described above surely includeprocesses to be performed in a time series manner in the describedorder, and include processes not necessarily performed in a time seriesmanner but in a parallel manner or individually.

The foregoing description of the embodiment of the present technology isin all aspects illustrative and not restrictive. It is understood thatnumerous other modifications and variations may be devised withoutdeparting from the scope of the present disclosure.

The present technology may be also in the following structures:

(1) An image processing apparatus, including:

an ambient-light image obtaining section configured to obtain anambient-light image in a first time range, the ambient-light image beingan image of an object captured with a predetermined exposure time;

a cumulative image generation section configured to generate acumulative image in a second time range after the first time range, thecumulative image being obtained by cumulative addition of each pixelvalue in a plurality of images, the plurality of images being of theobject captured one by one with the predetermined exposure time; and

a high-quality image generation section configured to generate ahigh-quality image, the high-quality image being obtained by subtractinga pixel value in the ambient-light image from a corresponding pixelvalue in a normalized image, the normalized image being the cumulativeimage normalized based on a total sum of the exposure time.

(2) The image processing apparatus according to (1), further including

a light turning-off detection section configured to determine whether ornot a lighting device is turned off, the lighting device being a lightsource different from a light source from which light is initiallyemitted for illumination of the object, wherein

when the lighting device is determined as being turned off, theambient-light image is captured.

(3) The image processing apparatus according to (2), in which thelighting device is turned on during imaging in a time range after theambient-light image is captured.

(4) The image processing apparatus according to (3), in which

the lighting device is held by a user, and is moved in an arc.

(5) The image processing apparatus according to (3), in which

the cumulative image generation section is configured to perform thecumulative addition of the images of the object captured in the secondtime range, the cumulative addition being performed by classifying theimages by direction based on information specifying toward whichdirections the lighting device emits light, and

the high-quality image generation section is configured to generateanother high-quality image by combining the high-quality images at apredetermined ratio, the high-quality images each being obtained bysubtracting the pixel value in the ambient-light image from thecorresponding pixel value in the normalized image, the normalized imagebeing each of the cumulative images classified by direction andnormalized based on the total sum of the exposure time.

(6) The image processing apparatus according to (5), further including

a display section configured to produce an image display, wherein

the display section is configured to display a GUI that is forspecifying the ratio of combining the plurality of high-quality images.

(7) The image processing apparatus according to any one of (1) to (6),in which

the cumulative image generation section is configured to divide thesecond time range into a plurality of short time ranges, and perform thecumulative addition of the images of the object captured in the secondtime range by classifying the images by the short time range,

and

the high-quality image generation section is configured to generateanother high-quality image by combining the high-quality images at apredetermined ratio, the high-quality images each being obtained bysubtracting the pixel value in the ambient-light image from thecorresponding pixel value in the normalized image, the normalized imagebeing each of the cumulative images classified by the short time rangeand normalized based on the total sum of the exposure time.

(8) The image processing apparatus according to (7), further including

a display section configured to produce an image display, wherein

the display section is configured to display a GUI that is forspecifying the ratio of combining the plurality of high-quality images.

(9) The image processing apparatus according to any one of (1) to (8),further including

a display section configured to produce an image display, wherein

in the second time range, the high-quality image is sequentiallydisplayed on the display section.

(10) The image processing apparatus according to (9), in which

a gain shows a gradual increase before a lapse of a predetermined timein the second time range, the gain being multiplied to a luminance valueof a pixel in the high-quality image displayed on the display section.

(11) The image processing apparatus according to (9), in which

a gain shows a gradual increase before a maximum luminance value of apixel in the cumulative image reaches a predetermined value, the gainbeing multiplied to a luminance value of a pixel in the high-qualityimage displayed on the display section.

(12) The image processing apparatus according to any one of (1) to (11),in which

in the cumulative image, a weight coefficient is multiplied to each ofthe pixel values in the plurality of images to prevent a per-imageproportional contribution of the images from being lower than apredetermined value, the images being captured in the second time range.

(13) The image processing apparatus according to (12), in which

occurrence of specular reflection on a surface of the object is detectedbased on a change of a pixel value in the cumulative image, and theweight coefficient is changed in value to reduce the proportionalcontribution of the image observed with the specular reflection.

(14) The image processing apparatus according to any one of (1) to (13),in which

by a predetermined computing process performed on the pixel values, alighting color is changed to illuminate the object in the imagescaptured in the second time range.

(15) An image processing method, including:

obtaining, by an ambient-light image obtaining section, an ambient-lightimage in a first time range, the ambient-light image being an image ofan object captured with a predetermined exposure time;

generating, by a cumulative image generation section, a cumulative imagein a second time range after the first time range, the cumulative imagebeing obtained by cumulative addition of each pixel value in a pluralityof images, the plurality of images being of the object captured one byone with the predetermined exposure time; and

generating, by a high-quality image generation section, a high-qualityimage, the high-quality image being obtained by subtracting a pixelvalue in the ambient-light image from a corresponding pixel value in anormalized image, the normalized image being the cumulative imagenormalized based on a total sum of the exposure time.

(16) A program causing a computer to function as an image processingapparatus, the apparatus including:

an ambient-light image obtaining section configured to obtain anambient-light image in a first time range, the ambient-light image beingan image of an object captured with a predetermined exposure time;

a cumulative image generation section configured to generate acumulative image in a second time range after the first time range, thecumulative image being obtained by cumulative addition of each pixelvalue in a plurality of images, the plurality of images being of theobject captured one by one with the predetermined exposure time; and

a high-quality image generation section configured to generate ahigh-quality image, the high-quality image being obtained by subtractinga pixel value in the ambient-light image from a corresponding pixelvalue in a normalized image, the normalized image being the cumulativeimage normalized based on a total sum of the exposure time.

What is claimed is:
 1. An image processing apparatus, comprising: anambient-light image obtaining section configured to obtain anambient-light image in a first time range, the ambient-light image beingan image of an object captured with a predetermined exposure time; acumulative image generation section configured to generate a cumulativeimage in a second time range after the first time range, the cumulativeimage being obtained by cumulative addition of each pixel value in aplurality of images, the plurality of images being of the objectcaptured one by one with the predetermined exposure time; and ahigh-quality image generation section configured to generate ahigh-quality image, the high-quality image being obtained by subtractinga pixel value in the ambient-light image from a corresponding pixelvalue in a normalized image, the normalized image being the cumulativeimage normalized based on a total sum of the exposure time.
 2. The imageprocessing apparatus according to claim 1, further comprising a lightturning-off detection section configured to determine whether or not alighting device is turned off, the lighting device being a light sourcedifferent from a light source from which light is initially emitted forillumination of the object, wherein when the lighting device isdetermined as being turned off, the ambient-light image is captured. 3.The image processing apparatus according to claim 2, wherein thelighting device is turned on during imaging in a time range after theambient-light image is captured.
 4. The image processing apparatusaccording to claim 3, wherein the lighting device is held by a user, andis moved in an arc.
 5. The image processing apparatus according to claim3, wherein the cumulative image generation section is configured toperform the cumulative addition of the plurality of images of the objectcaptured in the second time range, the cumulative addition beingperformed by classifying the plurality of images by direction based oninformation specifying toward which directions a lighting device emitslight, and the high-quality image generation section is configured togenerate another high-quality image by combining the high-quality imagesat a predetermined ratio, the high-quality images each being obtained bysubtracting the pixel value in the ambient-light image from thecorresponding pixel value in another normalized image, the othernormalized image being each of the cumulative images classified bydirection and normalized based on the total sum of the exposure time. 6.The image processing apparatus according to claim 5, further comprisinga display section configured to produce an image display, wherein thedisplay section is configured to display a GUI that is for specifyingthe ratio of combining the plurality of high-quality images.
 7. Theimage processing apparatus according to claim 1, wherein the cumulativeimage generation section is configured to divide the second time rangeinto a plurality of short time ranges, and perform the cumulativeaddition of the plurality of images of the object captured in the secondtime range by classifying the plurality of images by the short timerange, and the high-quality image generation section is configured togenerate another high-quality image by combining the high-quality imagesat a predetermined ratio, the high-quality images each being obtained bysubtracting the pixel value in the ambient-light image from thecorresponding pixel value in another normalized image, the othernormalized image being each of the cumulative images classified by theshort time range and normalized based on the total sum of the exposuretime.
 8. The image processing apparatus according to claim 7, furthercomprising a display section configured to produce an image display,wherein the display section is configured to display a GUI that is forspecifying the ratio of combining the plurality of high-quality images.9. The image processing apparatus according to claim 1, furthercomprising a display section configured to produce an image display,wherein in the second time range, the high-quality image is sequentiallydisplayed on the display section.
 10. The image processing apparatusaccording to claim 9, wherein a gain shows a gradual increase before alapse of a predetermined time in the second time range, the gain beingmultiplied to a luminance value of a pixel in the high-quality imagedisplayed on the display section.
 11. The image processing apparatusaccording to claim 9, wherein a gain shows a gradual increase before amaximum luminance value of a pixel in the cumulative image reaches apredetermined value, the gain being multiplied to a luminance value of apixel in the high-quality image displayed on the display section. 12.The image processing apparatus according to claim 1, wherein in thecumulative image, a weight coefficient is multiplied to each of thepixel values in the plurality of images to prevent a per-imageproportional contribution of the images from being lower than apredetermined value, the images being captured in the second time range.13. The image processing apparatus according to claim 12, whereinoccurrence of specular reflection on a surface of the object is detectedbased on a change of a pixel value in the cumulative image, and theweight coefficient is changed in value to reduce the proportionalcontribution of the image observed with the specular reflection.
 14. Theimage processing apparatus according to claim 1, wherein by apredetermined computing process performed on the pixel values, alighting color is changed to illuminate the object in the imagescaptured in the second time range.
 15. An image processing method,comprising: obtaining, by an ambient-light image obtaining section, anambient-light image in a first time range, the ambient-light image beingan image of an object captured with a predetermined exposure time;generating, by a cumulative image generation section, a cumulative imagein a second time range after the first time range, the cumulative imagebeing obtained by cumulative addition of each pixel value in a pluralityof images, the plurality of images being of the object captured one byone with the predetermined exposure time; and generating, by ahigh-quality image generation section, a high-quality image, thehigh-quality image being obtained by subtracting a pixel value in theambient-light image from a corresponding pixel value in a normalizedimage, the normalized image being the cumulative image normalized basedon a total sum of the exposure time.
 16. A non-transitory computerreadable medium having stored thereon, a set of computer-executableinstructions causing a computer to perform steps comprising: obtainingan ambient-light image in a first time range, the ambient-light imagebeing an image of an object captured with a predetermined exposure time;generating a cumulative image in a second time range after the firsttime range, the cumulative image being obtained by cumulative additionof each pixel value in a plurality of images, the plurality of imagesbeing of the object captured one by one with the predetermined exposuretime; and generating a high-quality image, the high-quality image beingobtained by subtracting a pixel value in the ambient-light image from acorresponding pixel value in a normalized image, the normalized imagebeing the cumulative image normalized based on a total sum of theexposure time.